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1.  Introduction 


The  general  objective  of  this  research  project  was  to  develop  and  to  experimentally  validate  a 
collaborative  assistant  for  rapid  knowledge  formation  and  reasoning.  This  assistant  will  enable  a 
team  of  Subject  Matter  Experts  (SME)  who  do  not  have  prior  knowledge  engineering  experience 
to  rapidly  construct,  update  and  extend  an  integrated  knowledge  base  for  a  complex  application. 
The  emphasis  of  this  research  was  on  acquiring  expert’s  problem  solving  knowledge  that  is  not 
normally  represented  in  written  documents.  This  task  is  complementary  to  that  of  acquiring 
knowledge  that  has  already  been  expressed  in  textbooks  or  other  documents.  This  research  has 
benefited  from  the  basic  research  done  under  the  AFOSR’s  Software  and  Systems  project  titled 
“Mixed-Initiative  Knowledge  Base  Development”  (2000-2003)  and  from  collaboration  with  the 
US  Army  War  College.  It  resulted  in  the  development,  experimental  use,  and  transition  of  a 
complex  knowledge  engineering  environment,  called  Disciple-Rapid  Knowledge  Formation 
(RKF),  and  its  application  to  the  military  Center  of  Gravity  (COG)  analysis  domain,  as  described 
in  the  following,  and  in  the  papers  from  section  10.1. 


Figure  1:  General  Approach  to  Rapid  Knowledge  Formation  by  Subject  Matter  Experts 

Figure  1  introduces  the  general  approach  investigated.  In  this  approach,  each  subject  matter 
expert  teaches  a  personal  Disciple-RKF  agent,  while  collaborating  with  it  in  solving  specific 
problems.  During  this  process,  the  Disciple-RKF  agent  learns  from  the  expert,  building, 
extending  and  improving  its  knowledge  base.  The  resulting  knowledge  bases  of  all  these 
Disciple-RKF  agents  are  then  integrated  by  a  knowledge  engineer.  The  Disciple-RKF  agent  with 
the  integrated  knowledge  base  can  then  be  used  in  three  ways.  It  can  be  used  by  a  non-expert  as  a 
problem  solver.  It  can  be  used  by  an  expert  as  a  problem  solving  assistant.  Finally,  it  can  be  used 
by  a  student  as  a  tutoring  system. 

2.  Synergistic  Collaboration  and  Transition  to  the  Army  War  College 

This  project  involved  a  multi-objective  collaboration  between  both  the  Teaming  Agents  Center 
of  George  Mason  University  and  the  Center  for  Strategic  Feadership  along  with  the  Department 
of  Military  Strategy,  Planning,  and  Operations  of  the  US  Army  War  College  (Tecuci  et  ah, 
2002a).  The  US  Army  War  College  provided  the  challenge  problem  (strategic  center  of  gravity 
analysis),  extensive  subject  matter  expertise  (faculty  and  high-ranking  officers  from  all  the 
military  services),  and  experimentation  support  for  both  the  developed  technology  and  the 
resulting  knowledge  bases  and  agents. 
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A  distinguishing  feature  of  this  collaboration  is  the  synergistic  integration  of  artificial 
intelligence  research,  with  military  strategy  research,  and  the  practical  use  of  agents  in  education, 
as  indicated  in  Figure  2. 
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Figure  2:  Synergistic  integration  of  artificial  intelligence  research  military  stratergy  research,  and  military 

education. 

The  artificial  intelligence  research  objective  was  the  development  of  knowledge  bases  and  agents 
by  subject  matter  experts  using  learning  agent  technology.  The  military  strategy  research 
objective  was  the  development  of  a  systematic  approach  to  center  of  gravity  determination.  The 
educational  objective  was  the  enhancement  of  the  educational  process  of  senior  military  officers 
and  strategic  leaders  through  the  use  of  intelligent  agent  technology.  This  integration  accelerated 
the  development  of  the  artificial  intelligence  technology  and  of  intelligent  agents  for  center  of 
gravity  analysis.  It  also  facilitated  the  transition  of  this  work  to  the  US  Army  War  College,  where 
Disciple  agents  have  been  used  since  2001  in  a  sequence  of  two  joint  warfighting  courses, 
“319jw  Case  Studies  in  Center  of  Gravity  Analysis,”  and  “589jw  Military  Applications  of 
Artificial  Intelligence.” 


3.  The  Center  of  Gravity  Analysis  Challenge  Problem 

Military  center  of  gravity  analysis  was  used  as  a  challenge  problem  to  test  the  knowledge 
acquisition,  learning,  and  problem  solving  methods  of  Disciple-RKF.  The  concept  of  center  of 
gravity,  introduced  by  Karl  von  Clausewitz  (Clausewitz,  K.V.  1832.  On  War,  translated  and 
edited  by  M.  Howard  and  P.  Paret.  Princeton,  NJ:  Princeton  University  Press,  1976),  is 
fundamental  to  military  strategy,  denoting  the  primary  source  of  moral  or  physical  strength, 
power,  or  resistance  of  a  force  (Strange,  J.  1996.  Centers  of  Gravity  &  Critical  Vulnerabilities: 
Building  on  the  Clausewitzian  Foundation  So  That  We  Can  All  Speak  the  Same  Language. 
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Quantico,  Virginia,  USA,  Marine  Corps  University).  The  most  important  objective  of  a  force 
(state,  alliance,  coalition,  or  group),  in  any  type  of  conflict,  is  to  protect  its  own  center  of  gravity 
while  attacking  the  center  of  gravity  of  its  enemy.  Therefore,  in  the  education  of  strategic  leaders 
at  all  U.S.  senior  military  service  colleges,  there  is  great  emphasis  on  center  of  gravity  analysis. 
This  analysis  requires  a  wide  range  of  background  knowledge  not  only  from  the  military  domain, 
but  also  from  the  political,  psychosocial,  economic,  geographic,  demographic,  historic, 
international,  and  other  domains.  In  addition,  the  situation,  the  adversaries  involved,  their  goals, 
and  their  capabilities  can  vary  in  important  ways  from  one  scenario  to  another.  Center  of  gravity 
analysis  is  a  very  good  example  of  knowledge-intensive,  expert  problem-solving  that  a  Disciple 
agent  should  be  able  to  learn. 


Our  approach  to  center  of  gravity  analysis,  based  on  the  work  of  Strange  (1996)  and  Giles  and 
Galvin  (Giles,  P.K.,  and  Galvin,  T.P.  1996.  Center  of  Gravity:  Determination,  Analysis  and 
Application.  CSL,  U.S.  Army  War  College,  PA:  Carlisle  Barracks),  and  developed  with  experts 
from  the  US  Army  War  College,  consists  of  two  main  phases,  identification  and  testing,  as 
shown  in  Figure  3.  During  the  identification  phase,  center  of  gravity  candidates  from  different 
elements  of  power  of  a  force  (such  as  government,  military,  people,  economy)  are  identified.  For 
instance,  a  strong  leader  is  a  center  of  gravity  candidate  with  respect  to  the  government  of  a 
force.  Then,  during  the  testing  phase,  each  candidate  is  analyzed  to  determine  whether  it  has  all 


the  critical  capabilities  that  are  necessary  to  be  the  center  of  gravity.  For  example,  a  leader  needs 
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Figure  3:  The  identification  and  testing  approach  to  center  of  gravity  analysis. 

to  be  protected,  stay  informed,  communicate  (with  the  government,  the  military,  and  the  people), 
be  influential  (with  the  government,  the  military,  and  the  people),  be  a  driving  force,  have 
support  (from  the  government,  the  military,  and  the  people),  and  be  irreplaceable.  For  each 
capability,  one  needs  to  detennine  the  existence  of  the  essential  conditions,  resources  and  means 
that  are  required  by  that  capability  to  be  fully  operative,  and  which  of  these,  if  any,  represent 
critical  vulnerabilities. 

4.  The  Architecture  of  the  Disciple-RKF  Agents 

The  architecture  of  Disciple-RKF  includes  the  components  from  Figure  4  (Boicu  et  al.,  2004). 
The  core  of  the  system  is  the  learning  agent  shell,  which  has  the  following  domain-independent 
components: 
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•  A  problem  solving  component  based  on  the  task  reduction  paradigm  of  problem  solving, 
including  the  following  modules: 

o  Modeling  assistant  that  helps  the  user  to  express  his/her  contributions  to  the 
problem  solving  process; 
o  Interactive  problem  solving  agent; 
o  Autonomous  problem  solving  agent. 

•  A  knowledge  acquisition  and  learning  component  for  acquiring  and  refining  the 
knowledge  of  the  agent  and  allowing  a  wide  range  of  operations  (including  ontology 
import,  user  definition  of  knowledge  base  elements,  ontology  learning,  and  rule  learning). 
This  component  includes  the  following  modules: 

o  Ontology  development  modules: 

■  Tree-based  browsers  for  objects  and  features; 

■  Graph-based  browsers  for  objects  and  features  (association  browser, 
hierarchical  browser); 

■  Viewers  and  Editors  for  objects  and  features; 

■  Ontology  import  module; 

■  Knowledge  base  merging  module. 

o  Instances  elicitation  modules: 

■  Scenario  elicitation  module; 

■  Scripts  editor. 

o  Learning  and  refining  modules: 

■  Task  formalization  and  learning  module; 

■  Explanation  generation  module; 

■  Rule  learning  module; 

■  Rule  refinement  with  positive  examples; 

■  Rule  refinement  with  negative  examples; 

■  Rule  analysis  module; 

■  Rule  regeneration  module; 

■  Exceptions-based  knowledge  base  refinement; 

■  Feature  learning  module. 

•  A  knowledge  base  manager  which  controls  the  access  to  and  the  updates  of  the 
knowledge  base.  Each  module  of  Disciple-RKF  can  access  the  knowledge  base  only 
through  the  functions  of  the  knowledge  base  manager. 

•  A  windows-based,  domain-independent,  graphical  user  interface. 

The  three  components  in  the  right  hand  side  of  Figure  4  are  the  typical  domain  dependent 
components  of  a  Disciple-RKF  agent  that  was  customized  for  a  specific  application,  such  as 
center  of  gravity  analysis.  The  Disciple-RKF/COG  agent  (also  referred  as  Disciple-COG) 
includes  the  following  components: 

•  A  customized  problem  solving  component  that  extends  the  basic  task-reduction 
component  in  order  to  satisfy  the  specific  problem  solving  requirements  of  the 
application  domain. 
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•  Customized  graphical  user  interfaces  which  are  built  for  the  specific  Disciple  agent  to 
allow  the  experts  and  the  end  users  to  communicate  with  the  agent  as  close  as  possible  to 
the  way  they  communicate  in  their  domains. 


•  The  knowledge  base  of  the  Disciple  agent  that  contains  knowledge  specific  to  the  center 
of  gravity  analysis  domain. _ 
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Figure  4:  The  Architecture  of  the  Disciple-RKF  Agents 
5.  Knowledge  Base  Development  Methodology 

The  Disciple  approach  covers  all  the  phases  of  agent  development  and  use  (Boicu  et  ah,  2001c; 
Tecuci  et  ah,  2002a,  2002b).  First,  a  knowledge  engineer  works  with  a  subject  matter  expert  to 
develop  an  ontology  for  the  application  domain.  They  use  the  ontology  import  module  (to  extract 
relevant  ontology  elements  from  existing  knowledge  repositories)  as  well  as  the  various  ontology 
editors  and  browsers  of  Disciple-RKF.  Figure  5  shows  the  interfaces  of  three  of  Disciple’s 
ontology  browsers:  at  top  is  the  association  browser  (which  displays  an  object  and  its 
relationships  with  other  objects),  at  bottom  left  is  the  tree  browser  (which  displays  the 
hierarchical  relationships  between  the  objects  in  a  tree  structure),  and  at  bottom  right  is  the 
hierarchical  browser  (which  displays  the  hierarchical  relationships  between  the  objects  in  a  graph 
structure). 

The  result  of  this  knowledge  base  development  phase  is  an  object  ontology  which  is  complete 
enough  to  be  used  as  a  generalization  hierarchy  for  learning,  allowing  the  expert  to  train  the 
Disciple  agent  on  how  to  solve  problems,  with  limited  assistance  from  a  knowledge  engineer. 
The  expert  formulates  a  specific  problem  solving  task  and  shows  the  agent  the  corresponding 
problem  solving  steps,  helping  the  agent  to  understand  them.  Each  problem  solving  step 
indicated  by  the  expert  consists  of  a  task  to  be  reduced,  a  question  related  to  that  task,  the  answer 
to  the  question,  and  one  or  several  subtasks  or  solutions  that  reduce  the  task.  The  top  part  of 
Figure  6  shows  a  fragment  of  a  reasoning  tree  (generated  by  the  interactive  problem  solver) 
consisting  of  a  sequence  of  task  reduction  steps.  The  bottom  part  shows  the  interface  of  the 
Modeling  Assistant  that  helps  the  expert  express  how  to  reduce  the  task  from  the  bottom  of  the 
reasoning  tree.  This  assistant  may  suggest  the  question  to  be  asked,  the  answer  to  the  question, 
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Figure  5:  Three  ontology  browsers  of  Disciple-RKF. 

and  the  knowledge  base  elements  the  expert  may  be  referring  to,  since  the  expert’s  input  is  in 
natural  language. 

Each  problem  solving  step  indicated  by  the  expert  is  an  example  from  which  the  agent  learns  a 
general  problem  solving  rule.  First,  the  expert  and  Disciple  collaborate  in  formalizing  the  tasks 
from  the  expert’s  examples  and  Disciple  learns  general  task  patterns.  Next  the  expert  helps 
Disciple  find  an  explanation  of  why  the  task  reduction  step  is  correct  and  Disciple  learns  a 
general  task  reduction  rule.  The  top  part  of  Figure  7  shows  the  interface  of  the  task  formalization 
module,  with  the  tasks  in  natural  language  in  the  middle  and  their  fonnalizations  in  the  right.  The 
bottom  part  of  Figure  7  shows  the  explanation  generation  and  selection  interface.  The  task 
reduction  rule  learned  from  the  example  specified  in  the  modeling  assistant  (see  bottom  of  Figure 
6)  is  shown  in  the  rule  viewer  from  Figure  8. 

As  Disciple  learns  new  rules  from  the  expert,  the  interaction  between  the  expert  and  Disciple 
evolves  from  a  teacher-student  interaction  toward  an  interaction  where  both  collaborate  in 
solving  a  problem.  During  this  mixed-initiative  problem  solving  phase,  Disciple  learns  not  only 
from  the  contributions  of  the  expert,  but  also  from  its  own  successful  or  unsuccessful  problem 
solving  attempts,  which  leads  to  the  refinement  of  the  learned  rules.  At  the  same  time,  Disciple 
extends  the  object  ontology  with  new  objects  and  features. 


6 


Figure  6:  The  reasoning  tree  and  the  modeling  assistant  interfaces. 

Copies  of  Disciple  agents  may  also  be  trained  in  parallel  by  different  experts.  In  this  case,  the 
individual  knowledge  bases  have  to  be  merged  into  an  integrated  knowledge  base,  as  discussed 
in  section  8. 
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Figure  7:  Task  formation  and  example  explanation. 

6.  Use  of  Disciple-COG  in  the  “Case  Studies  of  Center  of  Gravity  Analysis”  Courses: 
Scenario  Elicitation  Experiments 

Successive  versions  of  the  customized  Disciple-RKF/COG  agent  were  used  in  the  “Case  Studies 
in  Center  of  Gravity  Analysis”  course  at  the  Army  War  College  since  2001,  becoming  part  of  its 
regular  syllabus.  Figure  9  shows  how  the  agent  was  used.  First  Disciple  was  taught  how  to 
analyze  a  scenario,  based  on  the  expertise  of  the  course’s  instructor.  The  students  then  used 
Disciple  as  an  intelligent  assistant  that  helps  them  to  develop  a  center  of  gravity  analysis  of  a  war 
scenario.  Each  session  of  this  course  was  an  experiment  in  scenario  elicitation  from  subject 
matter  experts.  These  experiments  demonstrated  that  the  Disciple  approach  can  be  used  to 
develop  agents  that  have  been  found  to  be  useful  for  a  complex  military  domain  (Tecuci  et  ah, 
2004c). 


7.  Use  of  Disciple-RKF/COG  in  the  “Military  Applications  of  Artificial  Intelligence” 
Courses:  Agent  Teaching  Experiments 

Figure  10  shows  the  use  of  Disciple  in  the  “589jw  Military  Applications  of  Artificial 
Intelligence”  at  the  US  Army  War  College.  In  this  course,  the  students  teach  personal  Disciple 
agents  their  own  expertise  in  center  of  gravity  analysis  and  then  evaluate  both  the  developed 
agents  and  the  development  process.  In  the  2001  experiment,  the  students  used  historic  scenarios 
with  state  actors  (such  as  Okinawa  1943)  to  teach  personal  Disciple  agents  how  to  identify  center 
of  gravity  candidates.  In  the  2002  experiment,  the  students  used  historic  scenarios  and  a 
hypothetical  scenario  with  state  actors  to  teach  personal  Disciple  agents  how  to  identify  center  of 
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INFORMAL  STRUCTURE  OF  THE  RULE: 
IF 

T  BSt  whether  704  baa  701 . 
grtr? stlan-  Wfat  Is  a  mam  of  ? M  to  702  ? 

Answer:  703 
THEN 

T  eat  whether  703  which  protects  704  tea  any 
significant  vulnerability 


FORMAL  STRUCTURE  OF  THE  RULE: 
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gravity  candidates  and  to  eliminate  those  candidates  that 
do  not  pass  certain  tests.  In  the  2003  experiment,  the 
students  used  historic,  current  and  hypothetical  scenarios, 
with  both  state  and  non-state  actors,  to  teach  personal 
Disciple  agents  how  to  test  center  of  gravity  candidates 
based  on  the  concepts  of  critical  capabilities,  critical 
requirements  and  critical  vulnerabilities.  A  total  of  38  US 
and  international  officers  from  all  the  military  branches 
and  the  national  reserve  have  attended  these  courses.  At 
the  end  of  these  three  experiments,  10  of  them  strongly 
agreed,  20  agreed,  7  were  neutral  and  only  one  disagreed 
with  the  statement  “I  think  that  a  subject  matter  expert 
can  use  Disciple  to  build  an  agent,  with  limited  assistance 
from  a  knowledge  engineer.  ”  This  result  shows  that 
significant  progress  has  been  made  in  developing  the 
technology  that  will  allow  subject  matter  experts  to  build 
their  own  intelligent  assistants  (Tecuci  et  ah,  2004a). 


8.  Experiment  of  Parallel  Knowledge  Base 
Development  by  Subject  Matter  Experts 

The  Spring  2003  session  of  the  “Military  Applications  of 
Artificial  Intelligence”  course  included  an  experiment  in 
parallel  knowledge  base  development  by  subject  matter 
experts,  which  is  illustrated  in  Figure  1 1  (Tecuci  et  ah, 
2004b).  Before  starting  the  experiment,  the  Disciple -RKF 
agent  was  trained  to  identify  leaders  as  center  of  gravity 
candidates.  The  knowledge  base  of  this  agent  contained 
the  definitions  of  432  concepts  and  features,  29  tasks  and 
18  task  reduction  rules.  However,  the  agent  had  no 
knowledge  of  how  to  test  the  identified  candidates.  Then 


Figure  8:  Learned  Rule 
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Figure  9:  Training  and  using  the  Disciple  agent  (the  COG  course). 

a  domain  analysis  and  ontology  development  was  perfonned,  by  involving  all  the  subject  matter 
experts.  This  considered  the  example  of  testing  whether  Saddam  Hussein,  in  the  Iraq  2003 
scenario,  has  all  the  required  critical  capabilities  to  be  the  center  of  gravity  for  Iraq.  Based  on 
this  domain  analysis,  the  ontology  of  Disciple-RKF  was  extended  with  the  definition  of  37  new 
concepts  and  features  identified  with  the  help  of  the  subject  matter  experts. 
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The  13  subject  matter  experts  were  grouped  into  five  teams  (of  2  or  3  experts  each),  and  each 
team  was  given  a  copy  of  the  extended  Disciple-RKF  agent.  After  that,  each  team  trained  its 
agent  to  test  whether  a  leader  has  one  or  two  critical  capabilities,  as  indicated  in  Figure  1.  For 
instance,  Team  1  trained  its  agent  how  to  test  whether  a  leader  has  the  critical  capabilities  of 
staying  informed  and  being  irreplaceable.  The  training  was  done  based  on  three  scenarios  (Iraq 
2003,  Arab-Israeli  1973,  and  War  on  Terror  2003),  the  experts  teaching  Disciple-RKF  how  to 
test  each  strategic  leader  from  these  scenarios.  As  a  result  of  the  training  performed  by  the 
experts,  the  knowledge  base  of  each  Disciple-RKF  agent  was  extended  with  new  features,  tasks, 
and  rules,  as  indicated  in  Figure  11.  For  instance,  the  knowledge  base  of  the  agent  trained  by 
Team  1  was  extended  with  5  features,  10  tasks  and  10  rules.  The  average  training  time  per  team 
was  5  hours  and  28  minutes,  and  the  average  rule  learning  rate  per  team  was  3.53  rules/hour. 
This  included  the  time  spent  in  all  the  agent  training  activities  (i.e.,  scenario  specification, 
modeling  expert’s  reasoning,  task  formalization,  rule  learning,  problem  solving,  and  rule 
refinement). 


After  the  training  of  the  5  Disciple-RKF  agents,  their  knowledge  bases  were  merged  by  a 
knowledge  engineer,  who  used  the  knowledge  base  merging  tool  of  Disciple-RKF.  The 
knowledge  engineer  also  performed  a  general  testing  of  the  integrated  knowledge  base,  in  which 
the  acquired  10  new  features,  102  new  tasks,  and  99  new  rules  were  included.  During  this 
process  two  semantically  equivalent  features  were  unified,  4  rules  were  deleted,  and  12  other 
rules  were  refined  by  the  knowledge  engineer.  The  other  8  features  and  83  rules  acquired  from 
the  experts  were  not  changed.  Most  of  the  modifications  were  done  to  remove  rule  redundancies, 
or  to  specialize  overly  general  rules. 


/ 1 '  -Students  tea  dr 


tis'c  ip  It  their  C0( 
indy  sis  expertiu 
iin|j  sample 


using 
1  tjnarios  Drag  2fl( 


I  'Jar  on  terror  200: 
ftrabJsraeli  1973 


Figure  10:  Teaching  and  testing  a  personal  Disciple  agent  (the  MAAI  courage). 


Next,  each  expert  team  tested  the  integrated  agent  on  a  new  scenario  (North  Korea  2003),  and 
was  asked  to  judge  the  correctness  of  each  reasoning  step  performed  by  the  agent,  but  only  for 
the  capabilities  for  which  that  SME  team  perfonned  the  training  of  the  agent.  The  result  was 
98.15%  correctness. 
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Figure  11:  Experiment  of  rapid  knowledge  base  development  by  subject  matter  experts. 

9.  Conclusions  and  Future  Research  Directions 

The  experiment  described  in  section  8  and  illustrated  in  Figure  1 1  is  the  first  of  this  kind  ever 
performed,  showing  Disciple’s  capability  for  rapid  and  parallel  development  of  knowledge  bases 
by  subject  matter  experts,  with  limited  assistance  from  knowledge  engineers,  and  for  the 
integration  of  the  developed  knowledge  bases  into  a  functioning  agent.  However,  while 
significant  progress  has  been  made  in  developing  a  capability  of  rapid  knowledge  formation  by 
subject  matter  experts,  much  work  remains  to  be  done  to  improve  the  developed  methods.  For 
instance,  while  the  subject  matter  expert  has  an  increased  role  and  independence  in  the  agent 
development  process,  the  knowledge  engineer  still  has  a  critical  role  to  play.  The  knowledge 
engineer  has  to  assure  the  development  of  a  fairly  complete  and  correct  object  ontology.  The 
knowledge  engineer  also  has  to  develop  a  generic  modeling  of  the  expert’s  problem  solving 
process  based  on  the  task  reduction  paradigm.  Even  guided  by  this  generic  modeling,  and  using 
natural  language,  the  subject  matter  expert  has  difficulties  in  expressing  his  reasoning  process. 
Therefore  more  work  is  needed  to  develop  methods  for  helping  the  expert  in  this  task,  along  the 
path  opened  by  the  Modeling  Advisor.  The  performed  experimentations  have  also  revealed  that 
the  mixed-initiative  reasoning  methods  of  Disciple-RKF  could  be  significantly  empowered  by 
developing  the  natural  language  processing  capabilities  of  the  system. 

The  development  of  Disciple-RKF  has  emphasized  initial  knowledge  acquisition  from  a  subject 
matter  expert,  which  results  in  an  initial  knowledge  base.  Further  work  is  required  to  develop 
knowledge  acquisition  and  learning  methods  for  knowledge  base  extension  and  refinement,  as 
well  as  the  integration,  validation,  and  maintenance  of  the  knowledge  acquired  from  different 
subject  matter  experts.  In  particular,  the  Disciple  approach  can  naturally  be  extended  with 
methods  and  tools  for: 
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Knowledge  acquisition  for  knowledge  base  refinement  (including  rule  refinement  and 
ontology  learning)  using  mixed-initiative  and  multistrategy  techniques  that  will  exploit 
the  complementarity  between  human  and  automated  reasoning,  and  between  several 
learning  strategies  (such  as  learning  from  examples,  from  explanations,  by  analogy,  by 
abduction,  or  by  abstraction). 

Acquisition  of  meta -rules  that  will  capture  an  expert’s  rational  for  choosing  among 
different  ways  of  performing  a  problem  solving  task. 

Knowledge  bases  integration,  validation,  and  maintenance  (which  will  address 
problems  such  as  ontology  merging,  inconsistencies  within  a  knowledge  piece, 
redundancies  and  inconsistencies  among  knowledge  pieces  acquired  from  different 
experts,  refinement  and  reorganization  of  the  object  ontology  for  increased 
performance,  restructuring  and  refinement  of  the  acquired  problem  solving  rules,  etc.). 
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